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ABSTRACT 


Deesmsoelcveconsists OL a duscriminant analysis of 
composite marks for 23 selected categories in section B 
of the Marine Corps fitness report. The data for the study 
were taken from all the fitness reports on record for those 
officers in the grades of Captain, Major, and Lieutenant 
Colonel who appeared before promotion boards during FY8l. 

Discriminant scores were computed for all officers of 
a particular grade and those officers were then ranked 
peeemarmg FO Ehis score. It is shown that this ranking 
closely approximates a ranking by quality and so proves 
the discriminant score to be a viable performance index, 

a term whose definition and background are covered in the 
thesls. 

Pefeilvyemmene unique applicability of the discriminant 
analysis technique to the performance index problem is 
demonstrated. While generally unaffected by the distribu- 
tion of marks within a category, the weight assigned to 
each category in the discriminant function 1s very much 
influenced by the consideration given that category when 


the promotion decision is made. 
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fe eo NOD UGCTEON 


im this thesis, a technique is developed for construct- 
ing a performance index from the markings on section B of 
Biewiaerme Corps fitness report through the use of a 
Sederoet call tooOlscalled Discriminant Analysis. The teéech- 
ieee sts then applied to the markings on fitness reports 
Peo OLETcers Of the rank Captain through Lieutenant 
Colonel who appeared before promotion boards during the 
Poa leetscal year. 

The idea of a performance index is explored, criteria 
for measuring the effectivenss of such indices are sug- 
gested, and uses for the index--both current and envisioned-- 
are studied. In addition, past attempts at constructing a 
performance index are examined. 

The theory underlying the discriminant analysis tech- 
nigue is developed, not rigorously but in sufficient detail 
to allow readers to appreciate the applicability of the 
procedure and the significance and meaning of the results. 

Finally, the weights assigned to each of the fitness 
report categories in the discriminant function are examined. 
iiecmiaelwence Of Such factors as the distribution of marks 
within the categories on these weights is studied and the 
final conclusion lends considerable support to the use of 
Giscriminant analysis in the construction of the perform- 


Baeee index. 





Il. BACKGROUND 


A. THE MARINE CORPS FITNESS REPORT 

The USMC fitness report currently in use is NAVMC 
Bere 20835, Lt 1S in 3 sections. The first section, 
section A, records such administrative data as name, grade, 
Gurrent biliet and choices for further duty assignments. 
Section C provides space for a subjective word picture of 
the individual reported on. The second section, section B, 
is shown in Figure l. It consists of several fixed-choice 
markings on several categories or dimensions of perforn- 
ance. For the most part, there are 6 markings for each 
category. These markings are unsatisfactory, below 
average, above average, excellent and outstanding. There 
fama !lso@ a provision for a rating of "not observed" in case 
a particular performance dimension is not demonstrated 
Weelnegerne weriodsot the report. Block 15a has the same 
6 fixed-choice markings as the first group with an addi- 
tional block between each marking. Block 16 has a different 
scale of markings, aS can be seen in Figure l. 

Sewers thne mlocks in section B will not be considered 
Mimevemoelayes olocKkS 15b, 15c, l/, i8, 19 and 20. 

Throughout his career, each Marine officer is rated by 
means of the fitness report on several occasions. The 


rating occurs at least every 6 months or sooner if 
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occasioned by a promotion, transfer, change of commanding 
Orficer, etc. The fitness report is the principle means 
utilized by promotion boards in judging the qualifications 


of officers for advancement to the next higher grade. 


Be JHE PERFORMANCE INDEX 

The term performance index, in the context of this 
thesis, will mean some quantitative, composite measure 
aoeeezing an Officer's quality or promotability. It would 
indeed be difficult to separate these two attributes 
(quality or promotability) to any great extent, for in the 
Final analysis, it is the quality of the officer which 
(hopefully) results in the quality of his record and thence 
his advancement. At any rate, as will be seen in later 
chapters of this thesis, it will be the extent of the 
Sivilaciey OF an Officer's record to that of a group of 
promoted officers that will determine the measure of his 
quality--or his performance index. 

The composite nature of the performance index is 
stressed. Although there is certainly no substitute for 
the multi-dimensioned performance measure represented by 
all the categories of section B, together with all the 
other perhaps non-quantifiable indicators of quality con- 
sidered by a promotion board, there are certainly circum- 


stances where a single, composite measure of quality would 
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be useful and indeed, necessary. Some of those circum- 
Stances will be highlighted later on. 

In comparing and contrasting various performance 
indices (or indeed any method of performance appraisal) it 
1s necessary to establish and define the criteria to be 
used in determining the effectiveness or worth of each 
index or method. 

First, and perhaps foremost among the criteria that 
could be employed is the validity of the index in reflecting 
the true quality of the officer. It is very true that any 
variable describing an officer could be used as a perform- 
ance index--for example, hair color or date of birth--but 
it's doubtful that many of these types of indices could be 
described as valid. In practice, when judging the validity 
of a proposed index, the meaSurement of validity will be 
Beeietm@euitive and, possibly, quantitative. The intuitive 
judgement comes into play in rejecting the vast majority 
of possible indices (such as the two just mentioned). 

The quantitative measure of validity, when applied, 
will be in two phases. The first phase involves a measure 
that was actually employed at Headquarters, U.S. Marine 
Corps in the early months of this project. The measurement 
was made as follows: 

The proposed performance index was computed for all the 
officers whose records appeared before a particular promo- 


tion board. The officers were then ordered from highest to 
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Powest according to this computed index. The first n 
@eeeneecs On Enis list, where n was the actual number of 
officers who were promoted by the board, were chosen as 

a sample. The criterion measure was the sample proportion 
(p) of promoted officers. Although perhaps not a theore- 
tically sound measure, statistically, several characteristics 
of this measure were attractive and readily apparent. If 
the performance index utilized was a "perfect" indicator 

of promotability/quality, p should be 1.0. If the proposed 
index was in some sense a "poor" indicator, p should approx- 
imate the actual promotion proportion for all the officers 
who were considered. And a computed p that was less than 
the true promotion proportion would indicate that the index 
was in a sense negatively correlated with promotability. 
Again, it is admitted that the utility of this measure lies 
Memmi rts statistical grounding but in its computational 
ease, intuitive appeal and its ability to quickly and 
decisively eliminate some very bad candidates for a 
performance index. 

In Chapter 4, other quantitative criteria will be 
proposed and applied when our interest will not be ina 
"first-cut" measure designed to isolate good or bad ideas 
for indices but rather in investigating fine differences 
among several good indices in the form of discriminant 


FUnNeCtELONS. 
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Bememaos nOt as vital a criterion as validity, practi- 
cality has been de-emphasized as a result of the wide-spread 
use of high-speed computers in the administration and data- 
storage of the Marine Corps performance report system. Yet, 
Beeeeicality as a criterion still bears mention. 

Generally, the practicality of a performance index is 
a measure of, first, the accessibility of the information 
that is involved in the computation of the index and, 
second, the actual computational ease. In fact, the informa- 
tion used in computing all of the potential performance 
indices mentioned below was readily available in some sort 
of computer storage medium. Further, the differences in 
computational difficulty were practically immeasurable. 
However, the applicability of practicality as a viable 
criterion can be appreciated using a Simple example of a 
potentially highly accurate index wnose practicality is 
certainly questionable. Such an index would involve the 
reduction of the written appraisal in section C of the 
fitness report to a quantifiable measure. Indeed, the 
information is accessible but the computation has evident 
limitations in that attaching a single numerical score to 
a written evaluation on an individual would be a difficult 
Beiewwmecorsay the lleast, and would involve the resolution 


Seaowenissues as the proper scaling of such scores. 
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These criteria will now be applied to several indices 
that have either been tried in the past or envisioned for 
use inthe future. 

The idea of a performance index is by no means a new 
one. A form of index has been in use for some time by 
enlisted promotion boards in the Marine Corps. This index 
is a weighted, linear combination of selected markings 
from section B of the same report described in paragraph (a) 
above. At a typical promotion board session, the records 
of the candidates for promotion are distributed among the 
board members by performance index--to ensure that each 
member 1S given a representative sample of the candidate 
group and that no one member is forced to consider a group 
of predominantly below or above average candidates. Addi- 
tionally, the sheer number of records that each board 
member must consider dictates that he also use this index 
BemeGIrst-cut Criterion for promotion. That is, a candi- 
date with a relatively high index score is automatically 
promoted, with a low score automatically not promoted, and 
with the cases falling in between being more carefully 
examined before a final decision 1s made. 

The choice of selected markings and the weights assigned 
to each marking were determined quite subjectively over the 
years and represent no real concerted attempt at an optimum 
Spelcecoionmeor an Cptimum weighting, and therefore no real 


attempt at validity. 
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Pemecans Of distributing records among promotion board 
members, such as the process described above for the 
enlisted board, was one of the major potential uses envi- 
sioned for an officer's performance index. Another use of 
such an index could be in regression/correlation studies. 
For instance, the performance indices for a group of 
officers might be regressed on such variables as commis- 
slioning source, undergraduate degree, or officer candidate 
school standing to discover the degree of correlation 
between each of these factors and the officer's quality-- 
as measured by his performance index. Indeed, once such 
a regression function was constructed in this manner, an 
officer candidate's potential success should be well- 
estimated--certainly a valuable piece of information for 
eereemuiter. 

Among the first potential indices considered was a 
measure commonly known as the "truth teller". It was 
computed using a complicated formula involving the markings 
On items 15a, 15b, and 15c on section B of the fitness 
Beporee A Gombination of the number of officers ranked 
even, the number ranked above, and the number ranked below-- 
Sv voreguiethe fitness reports considered--gave a percentage 
grade, the truth teller. It was and, in some cases, Sie ome: 
is being utilized by some officer selection boards in much 


the same way as the index for enlisted boards--as a way of 
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distributing the records by quality among the members of 
the selection board. Its effectiveness as an index as 
measured by the criterion of practicality is indisputable. 
However, its accuracy can only be described as fair. 
Indeed, recent investigation uncovered several instances 

of misuse of this measure on the part of officers complet- 
ing reports on subordinates. As a result, in too many 
Saeeseene computed “truth teller" for an officer was an 
invalid measure of his quality. Not surprisingly, the 
quantitative criterion--the measure p, as described above-- 
when applied to this particular index yielded disappointing 
results. 

Another potential candidate for a performance index was 
the score on the military General Classification Test (GCT). 
The test 1s administered to every officer upon entry, is a 
permanent part of his military record, and so fares well 
with the practicality criterion. Yet, in no case was the 
Semomcea p on the quantitative accuracy criterion higher 
Epameenat for the truth teller. 

It was then decided to attempt to construct a composite, 
linear, weighted combination of the marking categories in 
section B to yield a performance index, much like the system 
in use by the enlisted promotion board. However, unlike 
the enlisted system which weights only selected marking 


categories, this scheme would consider all marking 
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categories (the only exceptions being those outlined in 
paragraph B above). The problem of assignment of weights 
to categories was resolved by the Delphi technique wherein 
a memo requesting suggested weighting schemes was circu- 
lated among several officers across as broad a spectrum of 
grades and expertise as possible. From these suggested 
segemes,;, a2 timal vector of category weights was fashioned. 
The effectiveness of this performance index as measured by 
Peemaccuracy Criterion p proved that this approach (the 
weighted linear combination) was a good one. The measured 
p for all three promotion boards was consistently between 
soouand .90Q. 

Yet there was still a basic flaw in both this scheme 
and to an even greater extent the one employed by the 
enlisted promotion boards. As was mentioned, the objective 
of the performance index was to present a composite score 
Pigieaalng the quality or promotability of a particular 
officer. The schemes presently in use represented the 
officer's relative standing among his peers when the cri- 
terion used might not necessarily represent his promota- 
bility in terms of what an actual promotion board action 
would reflect. Rather, these performance index scores 
reflected only the officer's quality measured against what 
a few selected questionnaire respondents thought quality 


snould be. 
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A solution seemed to be present in the fact that at 
the end of a board, two pieces of information were in hand. 
wee-tyeetic exact composition of the two groups (i.e., 
promoted group and not promoted group) was known. 
secondly, a multi-dimensional observation vector (repre- 
senting the markings in section B) was known for each 
pmerreer in each group. 

Taking advantage of this information, a possible 
alternative approach was seen to be as follows. 

Compute the mean values on each of the marking catego- 
ries for the promoted group and do the same for the not 
promoted group. Call this vector of mean values for each 
group the group centroid. A possible performance index 
for an individual, then, could be the extent of the simi- 
Pate veo an individual's markings to the promoted group 
eenceroid. 

Bas idea is basic to the discriminant analysis tech- 
nique and will be developed in some detail in the next 


chapter. 
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Iii. DISCRIMINANT ANALYSIS 


The following discourse on the discriminant analysis 
technique is intended primarily to enable the reader to 
appreciate the applicability of the technique to the 
construction of a performance index. The mathematics of 
the technique will be presented in a somewhat elementary 
fashion so as not to preclude anyone from gaining a full 
appreciation of the underlying ideas. For a more rigorous 
discussion, the interested reader is directed to any of 
Ene fererences listed in the bibliography, especially the 
book by Tatsuoka. 

Before beginning, it should also be mentioned that the 
discriminant analysis technique is applicable to the study 
of any number of groups greater than or equal to two. The 
discussion, however, will address primarily two-group 
analysis since the performance index problem and attendant 
data are both concerned with two groups--promoted and not 
Pmeoene@eea ©rricers. In addition, the derivation of the 
Gesceaitainanc function is much more straightforward in the 
case of two groups. In fact, Tatsuoka [Ref. 1] shows that 
two-group discriminant analysis is in many ways identical 
to multiple regression analysis. A rigorous discussion of 
the similarities (in a conceptual sense) between regression 
analysis and discriminant analysis can be found in Rulon 
Peet: 2). 
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ieee LaeORY 

Discriminant analysis is essentially a procedure for 
quantifying the differences between groups, albeit special 
groups. The underlying assumptions concerning each group 
are as follows: 

(1) Each group t (t = 1,2) has a number of members, each 
member being defined as a p-element vector of obser- 
vations or measurements, 

(2) The groups being investigated are separate and iden- 
tifiable and 

(3) The observations are assumed to have a multivariate 
normal distribution with equal covariance matrices. 
NOTE: further notational conventions that will be 

utilized throughout this chapter are as follows: 
- Bote is defined as the observation on the jth 
variable for the ith member of group t 
(Xi key a Seige Seay Reap! is therefore the p- 
element vector defining the ith member of 
GEcupmas 
Pee Sea Se. 3° eee Xe op) is the p-element vector 


of observed means on the p variables for group 


are 
Assumption (3) would appear to be rather restrictive 
especially in the context of the fitness report data that 
woliebe studied, given that many of the p variables will ex- 


hibit a highly negatively-skewed distribution. However, 
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Eisenbeis [Ref. 3] notes that there exists evidence that non- 
multivariate normal data may be used in a discriminant analy- 
Sis without significantly biasing the results. (But, as will 
be seen in the next chapter, the actual vectors of fitness 
report category marks that are entered into the discriminant 
analysis are composed of marks that represent means over 
several reports, and, as such, these marks tend to be more 
normally distributed than the original data). 

The study of the differences between groups entails a 
study of which particular elements or groups of elements in 
the p-element vectors define a dimension or direction along 
which the major group differences occur. This idea of cer- 
tain elements or groups of elements defining a dimension of 
greatest difference would logically entail an algebraic 
notion of linear combinations; specifically, a linear com- 
Pomadtlon Of the Original p observation variables that will 
somehow give a large difference between group means on that 
linear combination relative to the inherent within-group 
differences among values on that same combination. 

This linear combination of the original observation 
Variables is called the discriminant function and the coeffi- 
clients assigned to each of the original variables are called 
maeometsceriminant function coefficients. The discriminant 
function yields a discriminant score for the ith member of 


group t which is defined as: 


= | Ce re oe 
Sei Melee ue Oe ea eon ee 
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where Vir 1 = 1,2,...,p are the discriminant function coeffi- 
clients; en a? Mea aberene © Orlginal Observation 


variables for the ith member of group t; and Y is the 


wok 
discriminant score for that same member. As is evident, 
Sncesthis discriminant function with its coefficients is 
devised, each member of each group will have associated 
Mbeieice ad Giscriminant score. 

Peeqiece2stilustrates this notion for the special case 
Seep e= 42. In this Euclidean 2-space, each group member is 
located by a single dot and the collection of dots repre- 
senting One group is outlined. [In turn, the t-th group 


centroid is denoted by (X nine IGGOuUD Cemeroldmans 


ae 
Simply the 2 element vector representing the mean values of 
each original observation variable within a group. Within 
the restricted scope of this treatment, what will be of 
interest are the empirical distributions of the discriminant 
scores Y within each group, represented in the figure by 

a (Y)) and ave Phe emplercal@distributions reflect the 
projections of the original p observation variables onto 

the line L, where the projection is in fact the EY =; E> 
transformation accomplished by the discriminant function. 
Also of interest is the fact that given a particular orien- 
tation of L, the distance between the projections of the 
group centroids onto L iS maximized and the overlap petween 


the 2 groups is minimized. Intuitively, 1t would seem that 


Buewenmbentatiton of © should be such that it 1s parallel to 
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the line connecting the two group centroids in order that 
such a maximum centroid separation would occur. In most 
cases, this is indeed true. 

Discriminant analysis is the technique that will deter- 
iiemene OPLIMUumM discriminant coefficients and thus the 
optimum orientation of L so that indeed the differences in 
univariate means of the discriminant scores is maximized 


relative to the within-group variance in those scores. 


Beeb nw VATION OF THE DISCRIMINANT FUNCTION COEFFICIENTS 

The first step toward determining an optimum linear 
combination of the set of p variables such that the two 
group means will differ widely between themselves on that 
linear combination relative to the within-group differences 
on that linear combination is to suggest a criterion with 
which to measure "optimum". 

Once a linear combination of the p variables has been 
constructed, one is dealing with a single transformed varia- 
eweEnus, the familiar F-ratio for testing the signifi- 
cance of the difference between two group means on a single 
variable seems applicable. Indeed, it would seem that the 
optimum linear combination would yield a single variable 


that would maximize this F-ratio given by: 


SHE USS) b  (N-K) 


ss_/(N-K) eet Coo 


SS 





where N is the total number of members of both groups (ny of 
n, in this case), K is the number of Sroups (kK = 2 in this 
case), SS) is the sum of the squared deviations of each group 
mean around the grand mean, and SS. is the sum of the squared 


deviations of each group member's score around the group 


mean. That is, 


ae 1s the discriminant score for the ith member 


OE "Group ct, 


Me. iS seine gsoup t Gdiuseraminant score meant, 
Y,,. is the grand discriminant score mean and so, 
ron a ee 
Ss = Ge) Sn ne See (3-1) 
b wee 
and 
a 2 
SS | ee ees Meo (3-2) 
ba eee 
Since the second factor in the F-ratio, (N-K)/(K-1), is 


a constant for any given problem (where N and K are fixed), 
Eicertest Lactor SS) /SS,_, is the only essential quantity for 
measuring how widely a set of group means differ among them- 
selves relative to the amount of variability present within 
Eeeweroups. This ratio SS) /SS,, is what is called the dis- 
Samminame Criterion. Writing this discriminant criterion 


now in terms of the linear combination of original variables 
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instead of discriminant scores Y would involve making the 


Pow iewing Substitutions into equations 3-1 and 3-2: 


nem te Yo ee Tce * Vo*tip 
n 
: ip ‘ + 
= => ae + vox ew) 
te M32) Nicaea i=] P [ed J 
Ret iT ; 
ee = =( ec? Gaee e + ae Oe 
aerate = | Senate t=1 i=l P ttP 
Now maximizing the criterion over all vectors V = (Vy rVor 


ges will yield the optimum V which will be the vector 


of discriminant function coefficients we are seeking. 


Before continuing, mention should be made of the tech- 


nique employed by most commercial discriminant analysis 


packages--particularly the SPSS routine which was used to 


perform the bulk of the analysis in this study--in deriving 


Eaewerlscriminant function. 


the method outlined above 
Instead of expressing 
linear combination of the 


as in 


“rod: | aaa iT 


Uge sea 9 


BSsentialiiy sit 12S identical te 


With but a few notable exceptions. 
the Grscreimimnane Lumection as a 


Original variables X : x 


op Rg -c1e 1 om 


+ Vo*tip 
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the discriminant function is instead expressed as 


Zoe = te 

te a Ve oar: Bimisatst, 2 Vo“tip 
where Seis! i —e ieee Po mepmesents the standard deviations 
of the original variabl ae a ier fe 

g able Xe ig Bom 1tS group mean 126.5 44 
Pepecsents the standardized score). When these standardized 


Variables are substituted for the original variables in 
Seeetetons 3-1 amd 3=2, discriminant function coefficients 
can be derived to result in discriminant function values (dis- 
criminant scores) that are themselves in standard from. This 
means that, over all cases in the analysis, the discriminant 
scores will have a mean of 0 and standard deviation of l. 
The discriminant score of any one particular individual will 
represent the number of standard deviations that case iS away 
from the mean for all cases on the discriminant function. 

Morrison [Ref. 4] outlines several reasons why the stand- 
ardized values of the original variables are used in the 
analysis in place of the original variables, one of which is 
the fact that since our analysis is concerned with the "dis- 
tance" between two groups, and since statistical distance is 
normally measured in terms of standard deviations, one is 
justified in normalizing the original variables. 

The standardized discriminant coefficients (those com- 
puted using the standardized original variables) yleld a 


Fecaredeal Of information about the original variables. 
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When the sign is ignored, each coefficient represents the 
relative contribution of its associated variable to the dis- 
e-in@eant function score. (The interpretation is analogous 
to the interpretation of beta weights in multiple regression.) 
The sign merely indicates whether the contribution is positive 
or negative. One can even say that the greater the magni- 
euc@e OL a variable's coefficient, the more powerful that 
variable is in discriminating between the two groups. 

A question of real interest to the architect of any per- 
formance evaluation system is what statistics associated 
with a particular variable common to the two groups render 
One more discriminating than the other. As part of the 
analysis presented in Chapter V, relationships between the 
variable's coefficient magnitude and such statistics on that 
variable as within group variance, correlation and range and 


between-group F statistics will be examined. 


eee vet be SELECTION METHODS 

Referring again to the first few sentences of this chap- 
ter, recall that discriminant analysis requires that each 
group member be defined by a p-element vector of measurements 


x KP enak 


Saee@pservations. It was this vector (xX TE dea eres 


onli 
Waemuseaeto compute the discriminant function and that set 
of discriminant coefficients V = OY DNipyas 3 3 ON) such that 


the linear weighted combinations of the original p variables 


determined by the discriminant coefficients would have the 
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maximum separation between group means relative to the 
Wichai—-e@roup differences. 

In many instances, however, the full set of p variables 
may contain an excess of identical information about group dif- 
ferences, perhaps because of a high degree of correlation 
among certain of them. Or perhaps some of the variables 
may not be very useful in discriminating between the groups. 
It might therefore be useful to be able to determine a re- 
duced set of gq variables (q < p) that perform almost as well 
as the full set p in the discriminant analysis. 

Elsenbeis [Ref. 5] and Nie [Ref. 6] outline several such 
methods for reducing the dimension of the variable vector. 
Only one will be discussed now and indeed utilized in the 
next chapter. This particular method is the forward-selection 
procedure using the Wilks' Lambda statistic criterion. 

In a forward-selection procedure, the process begins by 
selecting the one variable which has the highest value on a 
particular criterion--in this case, the Wilks' Lambda which 
will be described in detail below. Next, the remaining p-l 
variables are paired one-at-a-time with the already-selected 
variable and the criterion is computed for each pair. The 
variable which, when paired with the first one selected, 
yields the best value on the criterion then becomes the sec- 
ond variable selected. These two variables are then combined 
with each of the p-2 remaining variables and the criterion 


then determines the third variable to be entered into the 


ore 





analysis. This process continues until either all the 
variables are selected or the inclusion of further variables 
yields less than a pre-determined improvement in the criter- 
10n Ineasure. 

At the same time, as variables are selected for inclu- 
Sion in the analysis, some of the variables previously se- 
lected may lose some of their discriminating power primarily 
because the information they contain might now be available 
in some other combination of the other variables. For this 
reason, before each step in the forward selection, all 
variables currently in the analysis are examined to see if 
Eaevyestill make a contribution to discrimination. If not, 
they are eliminated but are eligible to again enter ata 
later step. 

Sieesciend Variables are selected by this process, the 
discriminant analysis is performed using this q-vector of 
observations on each group member. 

In this study, the criterion for the stepwise selection 
is the Wilks' Lambda statistic. Wilks' Lambda, A, is de- 


fined as follows: 


| w | 


A = 
| T | 


Taenemvecic ©£ are the within groups and total sample sums 


Smemcouasessand cross—products (SSCP) matrices defined by 


Tatsuoka [Ref. 1], and |W| is the determinant of the matrix W. 
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fio examine several of the properties of Wilks' A, it is 


first helpful to see what A reduces to when there is but a 


single observation on each group member, i.e., p = l. 
In the case where p = 1, |W| reduces to 
2 ne = 9 
W = _-~- 
|W | ee eee ie) 


which is what SS. was Originally defined as (except that 


Y,; Was substituted for the X,,) and |T| reduces to 


2 oe — Z 
Mui = SS. + ) n(X.. = X..) 
W ier) ie 
em eel 
SS) 
where 
there are t = 2 groups, of sizes ns 


the ith member of group t is defined by Xvi (Simee 


there is only one variable characterizing each group 


member) and 


Ss ae er ey 
Ra) Sey ne, Caner a > ae 
ie ny ae ea N eee, a: 
Therefore, 
SS. 1 
mae = 1) aD EES Fas) 
SS. tg SS) 1 + (SS,/SS_) 





But note that the customary univariate F-ratio for a 


Simple analysis of variance (k-group) is given by 





Sp, (N-K) 
SS. (K-1) 

where N = ny t Nos and in this case, k = 2. Consequently, 
SS. N-K 


and, when substituting this expression for SS) /SS__ AES) 
3-3, we obtain 

, a i 

a TS e- -e ee 

Bron (3-4) we can see that, at least in the univariate 
case, there 1S an inverse relationship between A and F. In 
other words, the larger the disparity among several group 
neatuse,e relative to within-group variability, the larger is 
F, but the smaller is A. It is interesting and useful to 
fetemrtnat, according to Tatsuoka [Ref. 1], this relationship 
Ssewiorcas 1h the multivariate case (l1.e., p > l). 

At each step of the forward selection process using 
Wilks' A as the criterion, that variable which, when com- 
bined with the variables already in the analysis, yields 
the lowest value of A will be selected for inclusion. The 


process will continue until the inclusion of further variables 
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yields a decrease in A which is less than a predetermined 
value (chosen to be .001 in the analysis). These q varia- 
bles will then be entered into the discriminant analysis 


Beulecine . 


D. APPLICABILITY TO THE PERFORMANCE INDEX PROBLEM 

Refer, again, to Figure 1, the line L and the projec- 
tions of the original p-element vector of variables onto 
Poaeetime L--the projections being accomplished by the 
n° a ES transformation via the discriminant function. Re- 
Gall that this transformation yields a value called the 
discriminant score for each member of each group. The 
empirical distributions of those discriminant scores for the 
two groups were pictured as £, (¥) and oo. 

Refer now to Figure 3(a). The hypothetical empirical 
distributions of the discriminant scores for the officers 
who were selected (S) and not selected (NS) are pictured. 
Also pictured are the locations of the discriminant scores 
fomerour hypothetical officers (P11 P5/P3/Pq) whose records 
were not necessarily used in the analysis but whose military 
rank is identical to those officers whose records were. Now 
if we assume that the range of the discriminant scores was 
(-2,2), Figure 3(b) represents the individual discriminant 
scores for each member of the two groups against a scale to 
the right. The mean score for each group is again denoted 
(as it 1S in Figure 3(a)) by Y. Also pictured are the rela- 


PavTomleGcAatlons Of the discriminant scores for the four officers. 
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Recall that the transformation of the original p-vector 
of variables for each member has been accomplished such that 
the resultant discriminant scores show the greatest disparity 
between group means relative to within-group differences 
of those scores. 

Sme,ot the principal uses for discriminant analysis is 
in classifying cases that were not previously used in the 
analysis (hereinafter called fresh cases) into one of the k 
groups with a particular certainty of having made a correct 
Classification. The classification is made, in a sense, on 
the basis of the distance or similarity of the fresh case's 
vector of measurements to the group centroids of each of the 
Kygroups. Classification theory is a science in itself and 
is treated in some detail in several of the listed references. 
However, one 1S not too far wrong in saying that a fresh 
case should be classified as belonging to group k if the dis- 
tance between the discriminant score of the fresh case and 
the mean discriminant score for group k is less than the dis- 
tance between the discriminant score for the fresh case and 
any Other group mean score. For instance, it would appear 
that the officers labeled Py and P5 should be classified as 
belonging to the selected group whereas the officer labeled 
Py would more likely belong to the not-selected group. 

However, the intent of this thesis is to take this idea 
one step further and claim that, based on the computed dis- 


Griminant scores, the record of officer Py is in some sense 
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Detter@than the record of P5 Wheteh InwtGrn=is better than 

P3 which again is better than Py: This ordinal ranking can 
be rationalized in a sense because the discriminant score 
Emmott iecer Py is "further away" from the mean discriminant 
Be-mweewror the not selected officers than is the score for 
Sericer P>- At the same time, the score for officer Py is 
"Closer to" the mean score for the not selected officers than 
1s the score for officer Pa: andese forth. 

In fact, the hypothesis to be examined is that a ranking 
of the discriminant scores of officers of a particular grade 
from highest score to lowest is a legitimate method of rank- 
ing the quality of those same officers. Indeed, an officer's 
discriminant score would be his performance index. 

It 1s of course recognized that such a ranking by qual- 
ity has been based indirectly on factors that make up only 
Weeoreten Or the entire record on each officer. Totally 
ignored are such important performance dimensions as the type 
of duty performed during the period of a report, the written 
description on each officer contained in the report and 
other non-quantifiable points that every selection board 
rightfully takes into account when rendering its decision 
em eaen officer. 

However, bear in mind the uses to which such a performance 
index for officers would be put, as outlined in Chapter II. 
At no time would such an index be used in the actual promo- 


tion decision. At no time has a claim been made that a 
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performance index would be anything more than an "indicator" 
Severity, nmOt an actual measure. This particular index 
Seeeati1y fares well on the practicality criterion, and 
bears study as to its performance on a validity criterion, 


a study that will be undertaken in the next chapter. 
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IV. ANALYSIS AND RESULTS 


A. DATA 

The raw data set for the study was obtained from Head- 
quarters, Marine Corps personnel files. The set consisted 
6f all fitness reports on file for each officer who appeared 
before the Major, Lieutenant Colonel or Colonel promotion 
boards during the 1981 fiscal year. The information on file 
for each report was complete with the exception of the writ- 
ten appraisal in Section C. The name on each report was 
erased and the SSN encrypted to ensure anonymity. In all, 
there were 42,314 reports on 967 officers. 

The record on each fitness report consisted of the marks 
on the 23 performance categories in Section B of the report. 
However, instead of the markings that actually appear on the 
report (such as OS, EX, etc.), the marks on the record were 
encoded numerically as shown in Table I. 

The number of reports on each officer ranged from 25 to 
60 depending, of course, on his rank. To have entered each 
mark in each report aS a separate variable in the discrimi- 
nant analysis would have been impractical, so it was first 
decided to construct a single record for each officer. On 
this single record, the marks in each of the 23 categories 
would represent a composite of all the mmarks on that cate- 
Ben mover sall the reports on file. The method for construct— 


ing such a single record was an important decision, as well 
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TABLE I 


CODTNG=Or FIEINESS REPORT MARKS 


Fitness Report 


Marking 


Boc<s L3a to 14n 


Block 15a 


Block 16 


OS 
EX 
AA 
AV 
BA 
UN 


OS 
DGC 


Particularly Desire 
Be Glad 
Be Willing 


Prefer Not 
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Numerical 
Equivalent 


Oo FF WwW US WO 


OrRN WY Fk HN HW I DW LO 


OO ~I DW WH 





as a difficult one, for in choosing a scheme for computing 
a composite mark for each category, it was paramount to 
describe the tendencies of promotion board members. 

Conceivably, there are an infinite number of such schemes 
but, given resource constraints, only three were devised. 

It was thought that these three represented logital schemes 
that would accurately describe most board members behavior 
when screening an officer's reports. 

The first scheme involved computing a simple average (or 
mean) of the marks on all reports for a particular category 
as the composite mark for that category. The set of compo- 
Site category marks computed according to this scheme will 
be Known as et var iaBic set BN. 

The second scheme was to constrcut the composite mark 
for each category as a weighted average over all reports of 
the marks in that category. The weighting factor for each 
report's mark in a particular category would be the number 
of months covered in that report divided by the total number 
of months covered by all reports in the individual's file. 
For instance, 1f the total number of months covered by an 
individual's reports on file was 120 months, the weighting 
BaeEORNEOD a report of 12 months duration would be 0.1, ana 
one covering 6 months would be weighted .05. This variable 
set will be labeled WB. 

The third scheme was a variation on the second in that 


it too involved a weighted average of individual report 
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marks, but only over those reports written on an individual 
gmeais present grade. For instance, in the case of a Major 
appearing before the Lieutenant Colonels' promotion board, 
only those reports written on him while in the grade of 
Maj@n were considered. If all such reports covered a period 
SaeoGummontns, a report of 12 months duration would have a 
weighting factor of 0.2. This variable set will be titled 
GB. 

As was mentioned in a previous chapter, there would be 
considerable appeal in a reduced-dimension set of marks to 
be employed in computing an officer's discriminant score or 
performance index. For this reason, an additional subset 
of each of the original variable sets (BN, WB, and GB) was 
determined for each officer. The categories to be included 
in this subset were determined using the forward selection 
procedure and the Wilks' Lambda criterion described in Chap- 
ter III. The subsets determined in this manner will carry 
the same name as the original set from which the subset was 
taken, but with a suffixed -W (for example, BNW). 

During a preliminary examination of the data, it was 
found that, in certain particular categories of the fitness 
report, no marks had been assigned to that category on any 
of the reports on file (such a missing mark to be herein- 
after termed a missing value). As a result, any composite 
mark computed for that category would necessarily also exhi- 


bit a missing value. Unfortunately, packaged discriminant 
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edaweereorroucines willl totally exclude from the analysis 
the variable set of any officer that is not complete by 
Sveamene missing yalue. If, for one of the particular 
variable sets, a sizeable percentage of the officers 
exhibited missing values on one or more of the variables 
in that set, the set would of course be a poor candidate 
as a possible set of marks to be entered into the dis- 
criminant analysis, for two reasons: 

Beast, COMpPUtIng a discriminant £uUnction for a group 
uSing the variable sets from only a small sample of that 
group (with no guarantees that the sample is even remotely 
representative of the whole group) invites questionable 
results. Second, once such a discriminant function is 
determined, a valid discriminant score could be computed 
for only that same small sample of officers since only 
they would have a complete set of the requisite category 
marks by which the discriminant coefficients would be 
multiplied to yield a discriminant score. Herein lay one 
of the incentives in utilizing the forward-seiection pro- 
cedure to determine a reduced-dimension set of marks. 

The fewer the marks in the variable set, the smaller the 
percentage of officers who might exhibit a missing value 
On any one of the variables in that set. 

Conceivably, however, even such a reduced-dimension 

variable set might still contain variables on which a 


number of officers exhibit a missing value. 
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MmiesertOore, itt seemed’ a logical step to construct 
variable sets composed of variables for which it was known 
that a vast majority of officers had a valid mark. In an 
attempt to isolate the particular marking categories 
(variables) that were most often found to be missing or 
Metrmoeservyed On an officer's report, the distribution of 
marks for each of the marking categories (over all reports 
for all ranks) was examined. For each of the 23 marking 
categories, Table II shows the percentage of fitness 
reports on file on which that particular category was 
marked not observed or simply left blank. Constructing a 
variable set composed of variables for which few reports had 
missing values, and uSing this set in a discriminant analysis, 
would mean that close to 100% of the officer records would 
be used in the analysis. This in itself was reason enough 
to study the effectiveness of discriminant functions based on 
such sets. Therefore, for each of the three original sets 
(BN, WB, and GB), an additional subset was constructed com- 
posed of only those marking categories displaying less than 
a 30% missing value rate. These reduced variable sets carry 
the name of the original set from which they were formed, bdDut 
with a suffixed -R (e.g., BNR). 

In summary, the 12 variable sets to be used in the con- 
Struction of the discriminant functions to be analyzed in 
the following sectionsare listed--along with their descrip- 


tions--in Table III. 





TABLE If 


PERCENTAGE OF NOT OBSERVED OR MISSING MARKS 
FOR EACH CATEGORY 


% Not % Not 
Category Observed Category Observed 

misa 2A ela Zr 
3 You,5 *14g Plier 
oe dbs ie: 23.5 14h ee 
ILexal 32 a, *14i4 Pad ahs 
13e 537.6 7145 2 ore 
Ser 44.9 *14k 20.2 
13g elles = 20.4 
l4a 6o.2 14m sc alleres: 
*14b eas 1l4n 48.7 
=L4ac 1924 LSet eZ 
~i4a hes) 156 4.3 
*l4e We SG 


*used in reduced variable set 
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Wee tanle Set 


BN 


BNW 


BNR 


BNRW 


WBW 


WBR 


WBRW 


GB 


GBW 


TABLE III 


DESCRIPTION OF VARIABLE SETS 


Description 


mne jmarck On ecachweategqory via 
this set is the mean over all 
reports for that mark 


The variable set BN reduced 
by the forward selection 
technique 


Somitac CO Seto sN mouemonl, 
those marking categories 
asterisked in Table II are 
included 


The variable set BNR reduced 
by the forward selection 
technique 


The mark on each category in 
this set represents the length- 
weighted average over all 
reports for that mark 


The set WB reduced by the 
forward selection technique 


Simi lat ce: see WE, but only 
those marking categories 
asterisked in Table II are 
included 


The set WBR reduced by the 
forward selection technique 


Similar to set WB, but only 
those reports written while 
in an officer's present 
grade are considered 


The variable set GB reduced 
by the forward selection 
technique 
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TABGe Lit 
DESCRIPTION OF VARIABLE SETS 


(Continued) 


Variable Set Description 


GBR Similar to set GB, but only 
those marking categories 
asterisked in Table II are 
included 


GBRW The set GBR reduced by the 
forward selection technique 
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B. METHOD OF ANALYSIS 

Pies pcoblem of choosing that particular diseriminant 
Eunction yielding discriminant seores which best serve as 
Posteremance indices is in fact a problem of choosing that 
set of discriminating variables which, when used in the 
discriminant analysis, yields the best discriminant 
Funetion. Therefore, within each grade, discriminant 
functions computed using the 12 different variable sets 
will be subjected to several criteria with the intent of 
finding the optimum variable set for that grade, for this 
year's data. 

The criteria to be used will be as follows: 

een iinest criterion will be the qWantitative p criterion 
described in Chapter II and reiterated here. Given a 
discriminant function computed for a particular grade using 
one of the 12 variable sets, discriminant scores will be 
determined for the officers of that grade. whose records 
were entered in the analysis, using the computed discrimi- 
nant function. These officers will then be ordered by 
Piomewadrscktiminant score from highest to lowest. If, among 
these same officers, the actual number promoted 1s, say, Nn, 
then the proportion of promoted officers from the first n 
officers on the ordered list will be the criterion measure 
oy 

Mige secona criterion will be the proportion of officers 


of a particular grade whose variable sets were actually 
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entered into the analysis to obtain the discriminant 
function. This measure would also represent the proportion 
Peeerileers Of a particular grade for whom discriminant 
Peetecmeoula be computed using that discriminant function. 

It 1S recognized that there may exist other more 
elegant, perhaps more effective, criteria by which to 
measure differences among candidates. However, it must 
Peeoeee recognized that the two criteria mentioned must at 
least be included in any list of necessary measures. At 
any rate, the chapter will conclude by demonstrating that 
the "best" discriminant functions--those functions whose 
criterion measures were most satisfactory for each parti- 
cular grade--do indeed perform admirably in ranking the 
officers by quality, thus lending support to the two above- 
mentioned criteria as effectiveness measures and to the 
discriminant analysis technique as a method of constructing 
a performance index. 

(Throughout the next few sections, bear in mind that 
the records of captains would appear before a promotion 
board to major and the records of majors before a promotion 
board to Lieutenant Colonel, etc. Whenever results are 
Breesencted which involve promotion board action--for instance, 
the determination of an optimum variable set--the results 
will be under the name of the board. But whenever the 
results are to be applied to a particular grade of officer-- 


peemmeEnOodology OL computing discriminant scores tor tne 


0 





rank of Captain, for example--the results will be presented 


by rank). 


eee eeoULTS 
Ll. Majors' Board 

iMiemols> fitness reperts Of the 324 captains who 
appeared before the FY81 majors' promotion board were 
analyzed. Of the 324 captains in the analysis, 242 had 
in fact been selected for promotion for a rate of 74.7%. 
Tye results of the analysis for this board appear in 
maple IV. 

Many of the results found for the majors' board 
could have been predicted beforehand. For example, the 
officers in the grade of captain had, on the average, 
fewer reports on file than officers of the two higher 
grades. As a result, the incidence of missing values on 
the composite marks in the complete (non-reduced) variable 
sets would be substantial. The results substantiate this. 
As seen in Table IV, the discriminant function based on 
the variable set BN gives, by far, the best meaSure Of p. 
Yet Only 37% of the officers had records which were admis- 
Sible into the analysis. Of those variable sets reduced 
by the forward selection procedure, BNW gives the best 
combined performance on both criteria, yet still permits 


Only 21% of the records to enter the analysis. 


Sal: 





TABLE IV 


MAJORS' BOARD RESULTS 


Variable Set % Ee biting 

Entered into NO Missing 

Analysis Values 2] 
BN 37 yo Ske: 
BNW 81 eZ 
BNR 100 5 
BNRW 100 oO 
WB 36 woe 
WBW 81 yoon 
WBR 100 70S 
WBRW 100 7105 
GB 2 ~914 
GBW 45 A fase 
GBR 100 oo6 
GBRW 100 292 


*Actual promotion opportunity: 
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The results for the variable sets that were 
Peeeecad by restricting the marking categories based on 
percentage of missing values are also shown in Table IV. 
Pemeeeeurns Out, 100% of the officers’ records were admis- 
Sible into the analysis for all six of these variable 
sets. Of these sets, BNRW has the best measure in the p 
eiieerion. 

Summarizing, the fact that fully 100% of the 
captains’ records were used in the analysis when the 
variable set BNRW was used, coupled with the fact that 
the p criterion measure for BNRW is bettered only by 
variable sets allowing less than 40% of the records into 
the analysis, suggests that the set BNRW ought to be used 
eer ena s rank. 

2. Lieutenant Colonels' Board 

16,609 fitness reports for the 378 Majors appearing 
before the FY81 Lieutenant Colonels’ promotion board were 
analyzed. Of these 378 Majors, 254 were selected for 
promotion for a rate of 67.2%. The results for this board 
are shown in Table V. The best overall performance on the 
p criterion was obtained with a complete (non-reduced) 
variable set, GB. Yet only 18% of the records were used 
in the analysis. Again, 100% of the officers’ records 
were admissible in the case of the reduced sets (those 


sets whose categories were included by virtue of their 





TABLE V 


PEVCCLS “BOARD RESULTS 


Variable Set SS Jpguveshiackiere| 

Entered into NO Missing 

Analysis Values geo 
BN 7 mish ehe, 
BNW 99 .881 
BNR 100 .866 
BNRW 100 cisions 
WB 79 .874 
WBW 3) . 866 
WBR EO .858 
WBRW 100 FOGZ 
GB 18 TNS 
GBW 64 a Old 
GBR 100 sO 
GB RW 100 .846 

*Actual promotion opportunity: .6/2 
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missing value percentage). However, the set BNW, which 
allows fully 99% of the Majors' records to be used in the 
analysis, has a p criterion bettered only by the variable 
set GB which permits only 18% participation. Therefore, 
it seems that variable set BNW is the best for this particu- 
lar grade. 
SeecoOlonels, Board 

The results of the analysis on the Colonels' board 
are shown in Table VI. A total of 16,530 reports for 265 
officers were examined. Of these 265 officers, 143 were se- 
lected for a promotion rate of 54%. As was true for the 
Lieutenant Colonel's board, the GB variable set has the best 
measure on the p criterion, but a mere 19% of the records 
were entered into tne analysis. The differences between the 
remaining sets, however, are less distinctive. Set WBW has 
a full 99% of the records entering into the analysis, yet 
1ts p measure 1S not as good as the measure for set BN. 


In other words, there 1s no clear-cut choice to be made. 


Dae eolecLUuSIONS 

Especially in light of the results on the Colonels' 
Seaccawmers C§ILelcult tO rationalize making a decision on 
the best variable set for a particular rank without taking 
into account the results on the other ranks. Specifically, 
the results on both the Majors' and Lieutenant Colonels' 


poards suggest that variable sets whose category marks 


Do 





TABLE VI 


COLONELS —SOlRD RESULTS 


Variable Set Seeding 

Entered into NO Missing 

Analysis Values WD) 
BN 87 .846 
BNW 92 pork 
BNR 100 . 169 
BNRW 100 ac 
WB 87 HOoS 
WBW Sy) 5 Ole, 
WBR 100 Pe figsis 
WBRW 100 SE 
GB ILS, oe 
GBW 100 Fee alk 
GBR 100 ESS) 
GBRW 100 eS, 

“WGcugleorOnOrlon GQoportunity: 2.54 
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represent the simple averages over all reports of the marks 
POGeeati Category--Or SUbDSets Of this particular set--yield 
the best discriminant functions to be used in computing 
discriminant scores. This is fortunate since this variable 
set is, computationally, the least involved. So it would 
seem reasonable to add weight to the variable set BN in 

the Colonels' board analysis and, in so doing, choose it 

as the variable set to be used. 

Summarizing, then, the results on the three boards 
Suggest the following variables sets be used in the 
respective grades: 

For the grade of captain - set BNRW 
- For the grade of major - set BNW 
For the grade of Lieutenant Colonel - set BN 

The fact that the "completeness" of the optimum variable 
set for each rank (i.e., BN has more variables--1is more 
complete--than BNW) must increase with an increase in rank 
1s interesting. This point, among others, will be addressed 


in the following chapter. 


Eee te OTHESTZED MODEL 

From the beginning, the proposition has been made and 
Supported, at least in theory, that a ranking of officers 
by discriminant score would approximate a ranking by pro- 
Heraotetty. ANd, again, promotability is interpreted as 


quality for reasons that were outlined in Chapter II. In 


Sy 





Other words, given a list of officers ranked by discriminant 
score, the promotion probability of the officer who occupies 
the i-th position on the list should be greater than or 
equal to the promotion probability of the officer who 
Seecuptes the i-th + lst position, if the proposition is 
true. In more specific terms, assign a sequence number 

to each of the officers in the ordered list. The first 
officer on the list--the one with the highest discriminant 
score--would be assigned a sequence number of 1. The second 
highest on the list would have a sequence number of 2, and 
so forth. Then divide the officers in this ordered list 
into k sequential blocks with each block i (1i=l1,...,k) 
having n; Siticers. The first block would contain those 
officers with sequencenumbers 1 through ny the second 

+1 through n,+n 


al 2 


Bm@omeomeeoren. Count the number of officers in each group 1 


block would contain sequence numbers n 


Who were promoted, pr... Tien Ene orem@rion proportion for 
the i-th group would be pr,/n,. What the proposition would 
Suggest in this set-up is that the promotion proportion of 
each of the k groups is a decreasing function of the rela- 
tive placement of that group among the others. In other 
words, the promotion proportion of the i-th group, 

pr,/n; +25 should be greater than or equal to the propor- 


=Iotmotmeemcel=th + 1st group, PYi44/Ms 4 = Psat: 
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A simple k-sample chi-squared test for similar propor- 
tions among the k groups would, if rejected, imply that 
there is indeed a difference in the proportions but would 
suggest nothing else. 

Of interest in this case, however, is that, given 
there is a difference in group proportions, do those pro- 
portions vary in relation to, say, the mean sequence number 
of the groups? 

Fleiss [{Ref. 7] outlines a statistical technique de- 
Signed to investigate this last question. The technique is, 
in effect, a variation on the simple chi-square test for 
equal proportions. The hypothesis to be tested with this 
more detailed chi-square analysis, however, is that there 
is a Significant tendency for the group proportions to vary 
With mean group sequence number. Different methods of 
analysis are called for depending on how the proportions 
are hypothesized to vary, but only the simplest kind of 
variation will be considered nere--a linear one. 

That is, let D; again be the proportion of promoted 
officers in the i-th group and let x5 be the mean sequence 
number of the i-th group. The hypothesized model, then, 
giving the relationship between group proportion and group 


mean sequence number is: 


aie, 





where 8, the slope of the line, indicates the amount of 
change in the proportion per unit change in sequence number 
and a, the intercept, indicates the proportion expected 
when x = Q. 

The question may arise as to why a linear relationship 
1s being hypothesized when in fact the variation of D: with 
xX. May not be linear. What is of paramount interest is the 


a 


decreasing nature of the relationship, i.e., Pp, as a 
decreasing function of x. The actual shape of the function 
1s of secondary importance. But, hopefully, two things are 
eS : 

- First, the relationship--if not exactly linear--is 
Mermmcoo FameremOoved Lrom at, at least not far enough 
to result in rejection of the hypothesized model. 

- Secondly, if in fact the hypothesized model is rejected, 
it will be as a result of the actual shape of the 
function, and not because the function is not indeed 
decreasing. Other evidence (graphs) will, hopefully, 
Support this. 

At any rate, a negative sign for the slope estimate, b, 
would indicate that the function is indeed decreasing. The 
Significance of the "negativity" of this term can be 
tested, as will be seen. 

The test proceeds as follows: 


eee 


a number of officers in the i-th group 
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x. = mean sequence number of the ith group 
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Then the slope term, 8, is estimated by 


and the intercept term, a, is estimated by 


or, equivalently, 


a = p+ b(x, - x) 


ome 





“wn 


the expected proportion, Pio for each group can be calculated. 


The chi-square statistic is computed using 


me — ps) /p(l = p) 


>< 
ho 
iH 
fan 


1=1 
and has k-2 degrees of freedom (where k is the number of 
groups). 

The magnitude of this statistic will indicate the extent 
of the differences between the true proportions in each group 
(p;) and the expected proportions, proportions that would be 
expected if the linear relationship was true. A small value 
of the chi-square statistic would tend to support the linear 
model, a large value to reject it. 


Fleiss also shows that the following statistic, 


1s distributed chi-square with one degree of freedom and allows 
one to test the significance of the difference between the 
computed slope estimate, b, and 0. 

The test was run using the discriminant functions determined 
for each of the three ranks. A discriminant score was com- 
bueec fommecach Officer of that particular rank. The officers 


were then ranked from top to bottom by discriminant score and 
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arranged into sequential groups. Of course, the determination 
of each group size, n,, was dictated by the normal chi-square 
requirements such as fewer than 20% of the groups should have 
an expected number of promoted officers (n.p,) less than 5, 
etc. The results were as follows: 
1. Majors' Board 

For the grade of captain, a graphical representation 
of the relationship between P; and Xs is shown in Figure 4. 
Evident immediately is the general decreasing tendency of the 
group promotion proportions with an increase in group mean 
sequence number--a tendency that was expected. Also to be 
expected are the fluctuations in the proportions. The dis- 
criminant analysis technique, as has been mentioned many times 
before, takes into account only a small portion of the personal 
data that are used in the promotion board's decision. So a 
smooth curve--strictly decreasing--would certainly not be ex- 
pected. Due to the high overall promotion opportunity for 
the captains appearing before the major's board (74.7%), a 
Maer OuoLemOotlLonm pDProportion close to or at 1.0 for the first 
few groups could also have been predicted. This is also an 
explained departure from linearity--but in no way detracts 
from the effectiveness of the technique for this grade. 

The chi-square statistic for linearity is, for the 
captains' data, 10.25 which, with 4 degrees of freedom, is 
Significant at the .04 level. The association with sequence 


number of the proportion of officers promoted is thus not 
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precisely a linear one, but the departures from linearity 
can be explained, and in fact are not severe. Further, the 
slope term, which is of course negative, is significantly 
arererent from 0. 

2. Lieutenant Colonels' Board 

Figure 5 depicts the relationship of Ps with Ks for 
the grade of major. The general decreasing tendency of the 
relationship is again evident, but the departures from 
linearity in this case are much less notable due in part to 
the lower promotion rate for this grade (67.2%). 

Lending support to the graphical evidence are the 
results of the proportions test for the majors' data. The 
chi-square statistic was 1.805 which, with 6 degrees of 
freedom, was Significant at greater than the .9 level. The 
slope term, b, was again negative and significantly different 
iezeom 0 . 

3. The Colonels' Board 

Again, the general decreasing relationship of group 
proportions with mean group sequence number is evident from 
Figure 6. Two other characteristics of the curve in Figure 
3 could probably have been predicted. One, the relatively 
low overall promotion rate for this grade (54%) would result 
in a more rapid decrease in group promotion proportions with 
increase in sequence number in contrast to, say, the majors' 
board. Two, decisions on promotion to the grade of colonel 


im sole veistenumoer Of factors, few of them contained in 
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the 23 fitness report categories that, of necessity, were 
used by this analysis. As a result, the discriminant tech- 
nique is less effective as a performance index. This is evi- 
Gemced by the wide fluctuations in promotion proportions in 
wae Curve. 

Still, the chi-square statistic for linearity for 
this data was 9.354 which, with 7 degrees of freedom, was 
Significant at the .25 level. The slope term, again negative, 
Pasesigniiicicantly different from 0. So the hypothesized 
linear model is reasonable even at this grade. 

In summary, it can be said with a certain degree of 
confidence that the proposition concerning the discriminant 
technique as an effective method for constructing a performance 
index is supported. It has been shown that once a discrimi- 
nant function has been computed for officers of a certain grade 
using the applicable variable set, and these officers are then 
placed in an ordered list ranked by discriminant score, the 
promotion probability of an officer decreases in (generally 


linear) relation to his sequence number on that list. 


ae IMPLEMENTATION 

The implications of the supported preposition are as 
follows: 

taeemany Group OL Officers of a particular grade--not 
necessarily those officers whose records were used in the 
analysis. If discriminant scores were computed for those 


Smeuecers Using the discriminant function determined for that 
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Particular grade, and if these same officers were ranked on 

a list by discriminant score, then it's not unreasonable to 
claim that the promotion probability of an officer on that 
list (if he had conjecturally appeared before a board) would 
generally decrease with an increase in his sequence number on 
that list. And, as has been mentioned before, if one were 

tO Gquate promotability with quality, then the ordered list 
ranks those officers by quality--a basic requirement of a 
performance index. 

The analysis in this thesis has dealt exclusively with the 
three grades of Captain, Major, and Lieutenant Colonel. The 
analysis for the other grades, however, would be identical. 
The main thrust of the analysis, as was the case in this thesis, 
1s the determination of the variable set to be used in the 
discriminant routine. Once that is accomplished, the computa- 
Plone nrecdascriminant functions is faciliated by the widespread 
availability of discriminant analysis routines in commercial 
computer packageS--in particular, the SPSS package installed 
Greene system at HOMC, Washington, D.C. 

In the case of the three grades used in the analysis, the 
implementation of the results of this study go as follows: 

For an officer of the grade of Captain, Major, or Lieu- 
tenant Colonel, compute his applicable variable set according 
#6 the results contained in Section D. Compute his discrimi- 
nant score by multiplying the composite mark for each of the 


categories in the variable set by the corresponding coefficient, 
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found in Table VII, and then adding the constant, also found 
Piet meaole. s inethe case Of an officer of a particular rank 
who doesn't have a complete variable set of category marks 

and for whom, therefore, a discriminant score cannot be com- 
puted, the following procedures are suggested. When the dis- 
criminant scores are being utilized as a means of distributing 
record books among the members of a promotion board, a simple 
random procedure (e.g., alphabetically) for distributing the 
books of those officers without computed discriminant scores 
may be utilized. Efforts at estimating discriminant scores 
for those officers would not be simple and probably not worth- 
while simply because the vast majority of the books would have 
been distributed correctly, still ensuring each promotion 
board member a representative cross-section of the officers 
under consideration for promotion. When the discriminant 
scores are being utilized in regression studies (the use of 
discriminant scores in this context was addressed in Chapter 
TI, Section B), consideration in the analysis of only those 
officers with computed scores would probably have little or 
W@werbece On results. 

(The theoretical determination of the discriminant func- 
tion, presented in Chapter III, didn't involve the addition 
of a constant in the computation of discriminant scores. 
However, when discriminant scores are determined using raw 
(non-standardized) values for the composite category marks, 


the introduction of the constant is necessary to achieve 
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TABLE VII 


COMPUTATION OF DISCRIMINANT 


(Rank of Captain) 


Multiply the 
Composite Mark 
nOr 


l3a 


14b =e 


eae - 
l5a 3 


16 2 


GCOnscant -52. 


(Rank of Major) 
13d 
14a 
14c 


14d = 


l4e a 


L4f 


l4g ~-l. 
141i ae 
14n ules 
15a oF 


Constant -33. 
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(Ranikeot Ee Col) 


Multiply the 
Composite Mark 


i@ ie By 

isa -.5688985 
heya -.4045532 
ie FOS USS 
ieeye, 5 e8) A) lige) 9 
1l3e wo Lacie 
keha -.4781318 
IRE Ke| mo 7 Zee 
l4a 33101749 
14b =. ow ae 2 
1l4c .06260754 
14d SG SE 7 eA 
14e —Myo43235 
14f wos24529 
14g -.6931471 
14h . 1894585 
14i -.7334108 
143 ie 00 7 
14k 2.348909 
141 -.4552717 
14m = 6 (8) 25) syteibe i 
14n 25090 24 
15a org Taba ES, 
16 -4.149739 
Constant =e oo) ok 
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scores identical to those gotten using standardized data. 
And, as has been mentioned, the coefficients to be applied 
to raw data also differ from those to be applied to stand- 
ardized data.) 

In a sense, a discriminant function, which was computed 
based on the results of a particular board, reflects the 
views of that board--in that the prejudices shown by board 
members in considering certain marking categories more than 
others have a distinct effect on the final discriminant func- 
tion. It may be true that respectable results could be ex- 
pected using one particular function from year to year. How- 
ever, to ensure the "currency" of the functions--meaning that 
they are an accurate reflection of current board thinking, 
the functions ought to be recomputed for each grade following 
publication of the results of a board to select officers to 


the next higher grade. 
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Veo Ue eDeSCRIMINATORY POWER OF THE 
FITNESS REPORT CATEGORIES 
Pee LN TE RODUCTION 

Observing the discriminant function coefficients listed 
at the end of the previous chapter, one might reasonably be 
interested in what factors contribute to a certain fitness 
report category having a larger coefficient than another. 
Indeed, concerned might be the word since, in computing 
discriminant scores, the larger a category's coefficient, 
the more weight the mark on that category has in determining 
an individual's performance index. 

The fact that certain categories don't even appear in 
some of the discriminant functions 1S perhaps an even 
greater source of consternation, but rather easily ex- 
plained--at least more easily than the explanation con- 
cerning the relative size of the coefficients of the 
Gateg@erles included in the functions. 

First, recall that certain of the categories were 
excluded outright on several of the proposed variable sets 
Since a mark on those variables was missing on over 303% of 
the reports anyway. 

The exclusion of certain other of the variables was a 
result of the mechanics of the forward selection process. 
Briefly, if the discriminatory "information" provided by a 


category was contained in another category--or even ina 
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combination of two or more categories--already selected 
EPoomanalyors, itt wouldn't be included. Further details 
concerning such mechanics can be found in Eisenbeiss 
[Ref. 5]. High degrees of correlation between the cate- 
gories would make this effect even more severe. In the 
extreme, a correlation coefficient of 1.0 between all 
categories would result in only one category in the analysis. 

The correlations between several of the fitness report 
categories was in fact high, and this also interferes with 
the strict interpretation of a category's coefficient mag- 
nitude as the discriminatory power of that category--an 
interpretation espoused, with little or no explanation, by 
several authors. Most often, the interpretation goes as 
follows: using the standardized discriminant function 
coefficients, the more "discriminating" a variable is in 
distinguishing one group from another, the greater will be 
the magnitude of that variable's coefficient. 

But what is meant by "discriminating" and how is its 
degree to be measured? If an answer were to be found, 
steps could be taken to make the more important categories 
more discriminating and the magnitude of their associated 
coefficients greater--thereby increasing the influence of 
those categories in determining discriminant scores. 
Intuitively, several possible answers come to mind. 

First, it seems that certain statistics connected with 


the marking categories might have a correlation with 
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discrimination. For instance, the greater the variance 
or the range of marks in a category, the greater the 
possible distance between the group means on that category, 
and the larger, perhaps, the magnitude of its coefficient. 
A second intuitive answer, and certainly a desired one, 
is that the actions of the board in relying more heavily 
on the marks in certain categories in making their promo- 
tion decision should have an effect on the magnitude of the 
discriminant coefficients. 
Both possibilities will be briefly investigated in the 


following sections. 


bPyeeeen EGORY STATISTICS 

Rigorous proofs of the existence of hypothesized corre- 
lations between variable parameters and discriminant coeffi- 
cient magnitudes would be beyond the scope of this thesis-- 
1f indeed the correlations existed. 

The tack taken here, however, will be a simple empirical 
examination of the relationships--1if any--between certain 
statistics computed on the composite marks for each of the 
23 fitness report categories and the discriminant coeffi- 
cient computed on that same composite mark. This will be 
done for the data on each of the three grades of Captain, 
Major, and Lieutenant Colonel. The composite mark for 
each category will be tne simple average over all reports 


on file for that category. In other words, the variable 
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set BN will be entered into the discriminant analysis for 
each grade. Even though BN was not the recommended variable 
Fa ecOmeCLene Ome tide gqmades Captain Or Major, subsets of 
BN in fact were. In addition, a variable set that included 
all the categories was needed. The statistics to be 
examined are the following: 
- The average marks on each category for the promoted 
Grticers 
- The same average mark for the not promoted officers 
- The difference between the two means 
- The combined (promoted and not promoted officer records 
taken together) range of marks on each category 
- The combined sample variance of the marks in each 
caezeqory 
Meiiewi “ratio fer Gach Category mack. The F=ratlo to be 
defined as the ratio of the mean squares due to group 
means and the pooled within-group sample variance, as 
derimead in Chapter LIL, Section B. 

These statistics are displayed for all three grades in 
Table VIII. The results are interesting in themselves, but, 
unfortunately, there doesn't seem to be any clear-cut rela- 
tionship between coefficient magnitude and any of the 
Statistics. The only possible exception is the association 
of coefficient magnitude with the F-ratio. For example, 


over all three grades, the largest discriminant coefficient 
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and the largest F-ratio are associated with the same category. 
Plimenis Certainly isn't a surprising or a particularly use- 
ful result. For one thing, the statistics that determine 
ene F-ratio are a by-product of board action. The F-ratio 
is not associated with the category itself since a different 
grouping of officers into the promoted and not promoted 
groups would have resulted, perhaps, in a different mean 
square due to group means on the category and thence a dif- 
ferent F-ratio. Secondly, the computed discriminant coeffi- 
cients were produced in an attempt to find a linear combination 
of the original category marks that would give maximum separa- 
tion of group means on that linear combination relative to 
within-group variances on that combination. In a sense, this 
same group mean difference relative to within-group variation 
is what the F-ratio also expresses. So it would seem reason- 
able to expect a category having a large F-statistic to also 
contribute heavily to the above-mentioned linear combination-- 
i.e., have a large discriminant coefficient. 

However, as can be seen from Table VIII, those statistics 
which are independent of board action, namely range and vari- 
ance, seem to have no obvious correlation with coefficient 


magnitude. 


Gye rOsocliSbLe BFFECTS OF BOARD BIAS 
The second possible explanation for the different coeffi- 
cient magnitudes was the effect of board members' predisposi- 


tions in weighting certain categories more heavily when 
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Poecmenowthelr promotion decisions. That this may be so is 
suggested to a degree by the values in Table VIII. MTradi- 
tionally, such categories as "general value to the service" 
(item 15a), "leadership" (item 143), and "growth potential" 
(item 14n) have been among the most important and influen- 
tial on the fitness report. The coefficients for each of 
these categories are relatively large over all these grades. 
At the same time, one would expect the relative importance 
of certain of the categories to change, depending on the 
Peretecular board. For instance, "handling of officers" 
(item 13d) and "economy of management" (item 14m) are 
important measures of a Lieutenant Colonel's quality and 
perhaps to a lesser extent, a measure of a Captain's This 
Supposition is also supported by values in Table VIII. 

The effects of board bias on discriminant coefficients 
can also be demonstrated using a simple mathematical model. 
The analysis will exploit the following fact, covered in 
detail by Tatsuoka [Ref. l]. 

To each officer's standardized p-vector of category 
marks, attach a dependent, dichotomous variable (such as a 
variable taking values 1 or 0) representing the seer 
group. For instance, let 1 mean the officer was promoted 
and 0 mean not promoted. In a regression of this dichoto- 
mous variable on the p-vector of category marks, the com- 
puted regression weights will differ only by a multiplica- 


tive constant from the discriminant coefficients determined 
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by entering the p-vectors of category marks into a discrimi- 
nant analysis. 

so, to show that board bias affects discriminant coeffi- 
cients, it will suffice to show that the bias would effect 
the regression weights. 

Suppose that all officers before a particular board were 
characterized by marks on only two categories, A and B. That 
is, the ith officer's 2-vector of marks would be (a,b). 
Suppose further that the board decided to base their promo- 
tion decisions only on the mark for category A, completely 
1gnoring the B category mark. (Admittedly, this is an example 
of extreme, most improbable bias yet the example will serve 


to illustrate a point.) In addition, the marks on each of 


the categories represent standardized scores so that 
a0 and eves ee 
1 aL 


For simplicity, assume that the board considered only four 
officers, of whom two were promoted, and that all the pro- 
moted officers had a score of 1 for category A and the not 
promoted officers a score of -l. The scores on category B can 
take on any values, as long as they sum to 0. 

Attaching the dependent variable (1,0) to the applicable 


2-vectors of category marks, the data are: 


Dep variable Mark for A Mark for B 
Seticer al a Dy 
2 1 1, bo 
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: 0 al b 
= 0 aall b 


and the regression model becomes 
yy > x6 + e 


where 8 is the (2xl) matrix of regression weights, X is the 
feee2iemacrix of marks for A and B, Y is the (4x1) matrix of 
dependent variables (1,0), and e is the (4x1) matrix of error 
omIns . 

It now remains to show that the board's predisposition 
in favoring one category over the other will result ina 
regression weight for category A (8. ) large stidiie ciate 2o mr 
category B (8). 

The expression for the estimated regression weights 


ras 


g' = (8,785) is 


a =. 


B = (X'X) (X'Y) (where ' indicates transpose 


-l . 
and indicates inverse). 


Solving for each of the elements of 8, it can be shown 


that 


Fas 


det (X'X)8, > O 


and 


I! 
© 


det (X'X) 8, 


oT 





GiesLesults prove that the predisposition of the board 
in considering only the one category--A--in making the promo- 
tion decision had the effect of making the regression coeffi- 
cient for that category larger. Therefore, the discriminant 
Coefficient for the category A, if the four 2-vectors of 
category marks had been entered into a discriminant analysis, 
would also have been larger. It would also follow that the 
dependent variable score would be unaffected by B values, 
Since a change in B is multiplied by 0, and so the same would 
be true for the discriminant score. 

In the case where there are more than two categories, or 
where the consideration of categories is not so one-sided, 
the mathematics of the model are not nearly so tractable, but 
the results would be similar--i.e., the more consideration 


given a particular category when the promotion decision is 


meade, the larger the discriminant function coefficient for that 


Saeegqory. 


DB. CONCLUSION 

The conclusion to be drawn from the analysis above 1s 
particularly appealing and lends considerable credibility to 
the discriminant analysis technique as a method for construct- 


ing a performance index. 


There has been widespread and well-founded concern through- 


Siete aResMatane Corps regarding the lack of distribution in 


the category marks on the fitness report. With few exceptions, 


all of the marks on a typical report are distributed exclusively 
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@verwtrne excellent and outstanding blocks. A solution to the 
problem will have to soon be found but the options will surely 
all take time to implement. 

In the meantime, the discriminant analysis technique has 
shown its robustness even with this lack of distribution. 
Indeed, the range and variance of marks on a category seems 
empirically to have no effect on the coefficient for that 
Sategory in the discriminant function. 

What does have an effect is the inclination of the promo- 
tion board to rely more heavily on particular categories in 
rendering its promotion decision. In other words, a per- 
formance index determined by the discriminant function tech- 
nique will generally reflect the same standards the promotion 


board used in judging the quality of the officers it considered. 
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VI. SUGGESTIONS FOR FURTHER STUDY 


As has been mentioned, the main thrust of the analysis in 
Chapter IV was in the choice of variable set to be entered 
into the discriminant analysis. Although twelve sets were 
considered, the list of others that merit study is extensive. 
For instance, a set of composite category marks wherein a 
weight assigned to a particular report's mark is a decreasing 
function of age of the report (for example, weight = 1/(current 
date - report date)) might prove interesting. One of the 
important considerations in devising the different variable 
sets, however, is the logic of the weighting scheme employed. 
In other words, the scheme should be one that a promotion 
board member might be reasonably expected to employ when con- 
Sidering all the reports of a particular officer. With this 
in mind, the list of possible sets is probably not that 
extensive. 

Another area that might benefit from further study is the 
question of measures of effectiveness for the different vari- 
able sets. Those employed in this thesis were characterized 
by their simplicity and intuitive appeal. Perhaps others 
could be devised, however, which would more rigorously examine 
proposed variable sets. 

Still another area of interest would be the changes in 
discriminant weights over time. In other words, given a 


particular grade, would the discriminant weights for each 
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category remain generally constant when recomputed from year- 
to-year or would they change? 

The ultimate test of the effectiveness of the discrimi- 
nant technique would, of course, come when the discriminant 
scores computed for a particular promotion board were vali- 
dated with the data from the subsequent board. Specifically, 
if one were to rank the officers appearing before the next 
promotion board by discriminant scores based on the discrimi- 
nant function computed from this last board, would the results 
be supportive? That is, would the ranking by discriminant 
score again be generally a ranking by promotability, based on 
the subsequent board's promotion decisions? It is important 
that this be the case so the question would certainly be one 
worth investigating. 

The shift in category discriminant weights with the rank 
of the board was suggested and investigated to some extent 
in the previous chapter. Further study might reveal informa- 
tion suggesting, perhaps, different report formats for differ- 
ent grades. 

Finally, Amick [Ref. 8] suggests a method for determining 
the discriminant function which would exploit the high degree 
of correlation among several of the category marks. The 
method involves employing the Factor Analysis technique to 
first reduce the dimension of the variable set and then to 
SiEam@eEnowresultant factors into a discriminant routine. The 


reference outlines the advantages and disadvantages of such 


Sok 





a procedure. Its application to the performance index problem 


is evident and the method certainly bears study. 
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